{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Creating and Evaluating Solutions <a class=\"anchor\" id=\"top\"></a>\n",
    "\n",
    "In this notebook, you will train several models using Amazon Personalize, specifically: \n",
    "\n",
    "1. User Personalization - what items are most relevant to a specific user.\n",
    "1. Similar Items - given an item, what items are similar to it.\n",
    "1. Personalized Ranking - given a user and a collection of items, in what order are they most releveant.\n",
    "\n",
    "## Outline\n",
    "\n",
    "1. [Introduction](#intro)\n",
    "1. [Create solutions](#solutions)\n",
    "1. [Evaluate solutions](#eval)\n",
    "1. [Using evaluation metrics](#use)\n",
    "1. [Storing useful variables](#vars)\n",
    "\n",
    "## Introduction <a class=\"anchor\" id=\"intro\"></a>\n",
    "\n",
    "To recap, for the most part, the algorithms in Amazon Personalize (called recipes) look to solve different tasks, explained here:\n",
    "\n",
    "1. **User Personalization** - New release that supports ALL HRNN workflows / user personalization needs, it will be what we use here.\n",
    "1. **HRNN & HRNN-Metadata** - Recommends items based on previous user interactions with items.\n",
    "1. **HRNN-Coldstart** - Recommends new items for which interaction data is not yet available.\n",
    "1. **Personalized-Ranking** - Takes a collection of items and then orders them in probable order of interest using an HRNN-like approach.\n",
    "1. **SIMS (Similar Items)** - Given one item, recommends other items also interacted with by users.\n",
    "1. **Popularity-Count** - Recommends the most popular items, if HRNN or HRNN-Metadata do not have an answer - this is returned by default.\n",
    "\n",
    "No matter the use case, the algorithms all share a base of learning on user-item-interaction data which is defined by 3 core attributes:\n",
    "\n",
    "1. **UserID** - The user who interacted\n",
    "1. **ItemID** - The item the user interacted with\n",
    "1. **Timestamp** - The time at which the interaction occurred\n",
    "\n",
    "We also support event types and event values defined by:\n",
    "\n",
    "1. **Event Type** - Categorical label of an event (browse, purchased, rated, etc).\n",
    "1. **Event Value** - A value corresponding to the event type that occurred. Generally speaking, we look for normalized values between 0 and 1 over the event types. For example, if there are three phases to complete a transaction (clicked, added-to-cart, and purchased), then there would be an event_value for each phase as 0.33, 0.66, and 1.0 respectfully.\n",
    "\n",
    "The event type and event value fields are additional data which can be used to filter the data sent for training the personalization model. In this particular exercise we will not have an event type or event value. \n",
    "\n",
    "To run this notebook, you need to have run the previous notebooks, `01_Validating_and_Importing_User_Item_Interaction_Data` and `02_Validating_and_Importing_Item_Metadata.ipynb`, where you created a dataset and imported interaction and item metadata data into Amazon Personalize. At the end of that notebook, you saved some of the variable values, which you now need to load into this notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](img/03.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%store -r"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create solutions <a class=\"anchor\" id=\"solutions\"></a>\n",
    "[Back to top](#top)\n",
    "\n",
    "In this notebook, you will create solutions with the following recipes:\n",
    "\n",
    "1. User Personalization\n",
    "1. SIMS\n",
    "1. Personalized-Ranking\n",
    "\n",
    "The Popularity-Count recipe is the simplest solution available in Amazon Personalize and it should only be used as a fallback, so it will also not be covered in this notebook.\n",
    "\n",
    "Similar to the previous notebook, start by importing the relevant packages, and set up a connection to Amazon Personalize using the SDK."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "from time import sleep\n",
    "import json\n",
    "\n",
    "import boto3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Configure the SDK to Personalize:\n",
    "personalize = boto3.client('personalize')\n",
    "personalize_runtime = boto3.client('personalize-runtime')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In Amazon Personalize, a specific variation of an algorithm is called a recipe. Different recipes are suitable for different situations. A trained model is called a solution, and each solution can have many versions that relate to a given volume of data when the model was trained.\n",
    "\n",
    "To start, we will list all the recipes that are supported. This will allow you to select one and use that to build your model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'recipes': [{'name': 'aws-hrnn',\n",
       "   'recipeArn': 'arn:aws:personalize:::recipe/aws-hrnn',\n",
       "   'status': 'ACTIVE',\n",
       "   'creationDateTime': datetime.datetime(2019, 6, 10, 0, 0, tzinfo=tzlocal()),\n",
       "   'lastUpdatedDateTime': datetime.datetime(2020, 8, 25, 16, 57, 0, 148000, tzinfo=tzlocal())},\n",
       "  {'name': 'aws-hrnn-coldstart',\n",
       "   'recipeArn': 'arn:aws:personalize:::recipe/aws-hrnn-coldstart',\n",
       "   'status': 'ACTIVE',\n",
       "   'creationDateTime': datetime.datetime(2019, 6, 10, 0, 0, tzinfo=tzlocal()),\n",
       "   'lastUpdatedDateTime': datetime.datetime(2020, 8, 25, 16, 57, 0, 148000, tzinfo=tzlocal())},\n",
       "  {'name': 'aws-hrnn-metadata',\n",
       "   'recipeArn': 'arn:aws:personalize:::recipe/aws-hrnn-metadata',\n",
       "   'status': 'ACTIVE',\n",
       "   'creationDateTime': datetime.datetime(2019, 6, 10, 0, 0, tzinfo=tzlocal()),\n",
       "   'lastUpdatedDateTime': datetime.datetime(2020, 8, 25, 16, 57, 0, 148000, tzinfo=tzlocal())},\n",
       "  {'name': 'aws-personalized-ranking',\n",
       "   'recipeArn': 'arn:aws:personalize:::recipe/aws-personalized-ranking',\n",
       "   'status': 'ACTIVE',\n",
       "   'creationDateTime': datetime.datetime(2019, 6, 10, 0, 0, tzinfo=tzlocal()),\n",
       "   'lastUpdatedDateTime': datetime.datetime(2020, 8, 25, 16, 57, 0, 148000, tzinfo=tzlocal())},\n",
       "  {'name': 'aws-popularity-count',\n",
       "   'recipeArn': 'arn:aws:personalize:::recipe/aws-popularity-count',\n",
       "   'status': 'ACTIVE',\n",
       "   'creationDateTime': datetime.datetime(2019, 6, 10, 0, 0, tzinfo=tzlocal()),\n",
       "   'lastUpdatedDateTime': datetime.datetime(2020, 8, 25, 16, 57, 0, 148000, tzinfo=tzlocal())},\n",
       "  {'name': 'aws-sims',\n",
       "   'recipeArn': 'arn:aws:personalize:::recipe/aws-sims',\n",
       "   'status': 'ACTIVE',\n",
       "   'creationDateTime': datetime.datetime(2019, 6, 10, 0, 0, tzinfo=tzlocal()),\n",
       "   'lastUpdatedDateTime': datetime.datetime(2020, 8, 25, 16, 57, 0, 148000, tzinfo=tzlocal())},\n",
       "  {'name': 'aws-user-personalization',\n",
       "   'recipeArn': 'arn:aws:personalize:::recipe/aws-user-personalization',\n",
       "   'status': 'ACTIVE',\n",
       "   'creationDateTime': datetime.datetime(2019, 6, 10, 0, 0, tzinfo=tzlocal()),\n",
       "   'lastUpdatedDateTime': datetime.datetime(2020, 8, 25, 16, 57, 0, 148000, tzinfo=tzlocal())}],\n",
       " 'ResponseMetadata': {'RequestId': '59a4a7aa-4bb3-4b89-a223-eeaa203ab7a6',\n",
       "  'HTTPStatusCode': 200,\n",
       "  'HTTPHeaders': {'content-type': 'application/x-amz-json-1.1',\n",
       "   'date': 'Sun, 08 Nov 2020 06:46:11 GMT',\n",
       "   'x-amzn-requestid': '59a4a7aa-4bb3-4b89-a223-eeaa203ab7a6',\n",
       "   'content-length': '1259',\n",
       "   'connection': 'keep-alive'},\n",
       "  'RetryAttempts': 0}}"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "personalize.list_recipes()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output is just a JSON representation of all of the algorithms mentioned in the introduction.\n",
    "\n",
    "Next we will select specific recipes and build models with them.\n",
    "\n",
    "### User Personalization\n",
    "The User-Personalization (aws-user-personalization) recipe is optimized for all USER_PERSONALIZATION recommendation scenarios. When recommending items, it uses automatic item exploration.\n",
    "\n",
    "With automatic exploration, Amazon Personalize automatically tests different item recommendations, learns from how users interact with these recommended items, and boosts recommendations for items that drive better engagement and conversion. This improves item discovery and engagement when you have a fast-changing catalog, or when new items, such as news articles or promotions, are more relevant to users when fresh.\n",
    "\n",
    "You can balance how much to explore (where items with less interactions data or relevance are recommended more frequently) against how much to exploit (where recommendations are based on what we know or relevance). Amazon Personalize automatically adjusts future recommendations based on implicit user feedback.\n",
    "\n",
    "First, select the recipe by finding the ARN in the list of recipes above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "user_personalization_recipe_arn = \"arn:aws:personalize:::recipe/aws-user-personalization\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create the solution\n",
    "\n",
    "First you create a solution using the recipe. Although you provide the dataset ARN in this step, the model is not yet trained. See this as an identifier instead of a trained model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "user_personalization_create_solution_response = personalize.create_solution(\n",
    "    name = \"personalize-poc-userpersonalization\",\n",
    "    datasetGroupArn = dataset_group_arn,\n",
    "    recipeArn = user_personalization_recipe_arn\n",
    ")\n",
    "\n",
    "user_personalization_solution_arn = user_personalization_create_solution_response['solutionArn']\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\"arn:aws:personalize:us-east-1:835319576252:solution/personalize-poc-userpersonalization\"\n"
     ]
    }
   ],
   "source": [
    "print(json.dumps(user_personalization_solution_arn, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create the solution version\n",
    "\n",
    "Once you have a solution, you need to create a version in order to complete the model training. The training can take a while to complete, upwards of 25 minutes, and an average of 90 minutes for this recipe with our dataset. Normally, we would use a while loop to poll until the task is completed. However the task would block other cells from executing, and the goal here is to create many models and deploy them quickly. So we will set up the while loop for all of the solutions further down in the notebook. There, you will also find instructions for viewing the progress in the AWS console."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "userpersonalization_create_solution_version_response = personalize.create_solution_version(\n",
    "    solutionArn = user_personalization_solution_arn\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"solutionArn\": \"arn:aws:personalize:us-east-1:835319576252:solution/personalize-poc-userpersonalization\",\n",
      "  \"ResponseMetadata\": {\n",
      "    \"RequestId\": \"20623f71-407c-46ab-80bd-3d6a9dacb797\",\n",
      "    \"HTTPStatusCode\": 200,\n",
      "    \"HTTPHeaders\": {\n",
      "      \"content-type\": \"application/x-amz-json-1.1\",\n",
      "      \"date\": \"Sun, 08 Nov 2020 06:46:11 GMT\",\n",
      "      \"x-amzn-requestid\": \"20623f71-407c-46ab-80bd-3d6a9dacb797\",\n",
      "      \"content-length\": \"105\",\n",
      "      \"connection\": \"keep-alive\"\n",
      "    },\n",
      "    \"RetryAttempts\": 0\n",
      "  }\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "userpersonalization_solution_version_arn = userpersonalization_create_solution_version_response['solutionVersionArn']\n",
    "print(json.dumps(user_personalization_create_solution_response, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### SIMS\n",
    "\n",
    "\n",
    "SIMS is one of the oldest algorithms used within Amazon for recommendation systems. A core use case for it is when you have one item and you want to recommend items that have been interacted with in similar ways over your entire user base. This means the result is not personalized per user. Sometimes this leads to recommending mostly popular items, so there is a hyperparameter that can be tweaked which will reduce the popular items in your results. \n",
    "\n",
    "For our use case, using the Movielens data, let's assume we pick a particular movie. We can then use SIMS to recommend other movies based on the interaction behavior of the entire user base. The results are not personalized per user, but instead, differ depending on the movie we chose as our input.\n",
    "\n",
    "Just like last time, we start by selecting the recipe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "SIMS_recipe_arn = \"arn:aws:personalize:::recipe/aws-sims\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create the solution\n",
    "\n",
    "As with HRNN, start by creating the solution first. Although you provide the dataset ARN in this step, the model is not yet trained. See this as an identifier instead of a trained model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"solutionArn\": \"arn:aws:personalize:us-east-1:835319576252:solution/personalize-poc-sims\",\n",
      "  \"ResponseMetadata\": {\n",
      "    \"RequestId\": \"a27bfa79-c34c-4b85-890c-a3b2652843ad\",\n",
      "    \"HTTPStatusCode\": 200,\n",
      "    \"HTTPHeaders\": {\n",
      "      \"content-type\": \"application/x-amz-json-1.1\",\n",
      "      \"date\": \"Sun, 08 Nov 2020 06:46:11 GMT\",\n",
      "      \"x-amzn-requestid\": \"a27bfa79-c34c-4b85-890c-a3b2652843ad\",\n",
      "      \"content-length\": \"90\",\n",
      "      \"connection\": \"keep-alive\"\n",
      "    },\n",
      "    \"RetryAttempts\": 0\n",
      "  }\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "sims_create_solution_response = personalize.create_solution(\n",
    "    name = \"personalize-poc-sims\",\n",
    "    datasetGroupArn = dataset_group_arn,\n",
    "    recipeArn = SIMS_recipe_arn\n",
    ")\n",
    "\n",
    "sims_solution_arn = sims_create_solution_response['solutionArn']\n",
    "print(json.dumps(sims_create_solution_response, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create the solution version\n",
    "\n",
    "Once you have a solution, you need to create a version in order to complete the model training. The training can take a while to complete, upwards of 25 minutes, and an average of 35 minutes for this recipe with our dataset. Normally, we would use a while loop to poll until the task is completed. However the task would block other cells from executing, and the goal here is to create many models and deploy them quickly. So we will set up the while loop for all of the solutions further down in the notebook. There, you will also find instructions for viewing the progress in the AWS console."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "sims_create_solution_version_response = personalize.create_solution_version(\n",
    "    solutionArn = sims_solution_arn\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"solutionVersionArn\": \"arn:aws:personalize:us-east-1:835319576252:solution/personalize-poc-sims/666f373e\",\n",
      "  \"ResponseMetadata\": {\n",
      "    \"RequestId\": \"b4a6c1c1-6223-4cc1-9153-3da6606191fd\",\n",
      "    \"HTTPStatusCode\": 200,\n",
      "    \"HTTPHeaders\": {\n",
      "      \"content-type\": \"application/x-amz-json-1.1\",\n",
      "      \"date\": \"Sun, 08 Nov 2020 06:46:12 GMT\",\n",
      "      \"x-amzn-requestid\": \"b4a6c1c1-6223-4cc1-9153-3da6606191fd\",\n",
      "      \"content-length\": \"106\",\n",
      "      \"connection\": \"keep-alive\"\n",
      "    },\n",
      "    \"RetryAttempts\": 0\n",
      "  }\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "sims_solution_version_arn = sims_create_solution_version_response['solutionVersionArn']\n",
    "print(json.dumps(sims_create_solution_version_response, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Personalized Ranking\n",
    "\n",
    "Personalized Ranking is an interesting application of HRNN. Instead of just recommending what is most probable for the user in question, this algorithm takes in a user and a list of items as well. The items are then rendered back in the order of most probable relevance for the user. The use case here is for filtering on unique categories that you do not have item metadata to create a filter, or when you have a broad collection that you would like better ordered for a particular user.\n",
    "\n",
    "For our use case, using the MovieLens data, we could imagine that a VOD application may want to create a shelf of comic book movies, or movies by a specific director. We most likely have these lists based title metadata we have. We would use personalized ranking to re-order the list of movies for each user, based on their previous tagging history. \n",
    "\n",
    "Just like last time, we start by selecting the recipe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "rerank_recipe_arn = \"arn:aws:personalize:::recipe/aws-personalized-ranking\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create the solution\n",
    "\n",
    "As with the previous solution, start by creating the solution first. Although you provide the dataset ARN in this step, the model is not yet trained. See this as an identifier instead of a trained model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"solutionArn\": \"arn:aws:personalize:us-east-1:835319576252:solution/personalize-poc-rerank\",\n",
      "  \"ResponseMetadata\": {\n",
      "    \"RequestId\": \"bcd3deaa-07e5-47ba-bbd3-abc41b61bb57\",\n",
      "    \"HTTPStatusCode\": 200,\n",
      "    \"HTTPHeaders\": {\n",
      "      \"content-type\": \"application/x-amz-json-1.1\",\n",
      "      \"date\": \"Sun, 08 Nov 2020 06:46:12 GMT\",\n",
      "      \"x-amzn-requestid\": \"bcd3deaa-07e5-47ba-bbd3-abc41b61bb57\",\n",
      "      \"content-length\": \"92\",\n",
      "      \"connection\": \"keep-alive\"\n",
      "    },\n",
      "    \"RetryAttempts\": 0\n",
      "  }\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "rerank_create_solution_response = personalize.create_solution(\n",
    "    name = \"personalize-poc-rerank\",\n",
    "    datasetGroupArn = dataset_group_arn,\n",
    "    recipeArn = rerank_recipe_arn\n",
    ")\n",
    "\n",
    "rerank_solution_arn = rerank_create_solution_response['solutionArn']\n",
    "print(json.dumps(rerank_create_solution_response, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create the solution version\n",
    "\n",
    "Once you have a solution, you need to create a version in order to complete the model training. The training can take a while to complete, upwards of 25 minutes, and an average of 35 minutes for this recipe with our dataset. Normally, we would use a while loop to poll until the task is completed. However the task would block other cells from executing, and the goal here is to create many models and deploy them quickly. So we will set up the while loop for all of the solutions further down in the notebook. There, you will also find instructions for viewing the progress in the AWS console."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "rerank_create_solution_version_response = personalize.create_solution_version(\n",
    "    solutionArn = rerank_solution_arn\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"solutionVersionArn\": \"arn:aws:personalize:us-east-1:835319576252:solution/personalize-poc-rerank/44c934a6\",\n",
      "  \"ResponseMetadata\": {\n",
      "    \"RequestId\": \"61ffe655-1749-49f6-9a67-3aea32515ba8\",\n",
      "    \"HTTPStatusCode\": 200,\n",
      "    \"HTTPHeaders\": {\n",
      "      \"content-type\": \"application/x-amz-json-1.1\",\n",
      "      \"date\": \"Sun, 08 Nov 2020 06:46:12 GMT\",\n",
      "      \"x-amzn-requestid\": \"61ffe655-1749-49f6-9a67-3aea32515ba8\",\n",
      "      \"content-length\": \"108\",\n",
      "      \"connection\": \"keep-alive\"\n",
      "    },\n",
      "    \"RetryAttempts\": 0\n",
      "  }\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "rerank_solution_version_arn = rerank_create_solution_version_response['solutionVersionArn']\n",
    "print(json.dumps(rerank_create_solution_version_response, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### View solution creation status\n",
    "\n",
    "As promised, how to view the status updates in the console:\n",
    "\n",
    "* In another browser tab you should already have the AWS Console up from opening this notebook instance. \n",
    "* Switch to that tab and search at the top for the service `Personalize`, then go to that service page. \n",
    "* Click `View dataset groups`.\n",
    "* Click the name of your dataset group, most likely something with POC in the name.\n",
    "* Click `Solutions and recipes`.\n",
    "* You will now see a list of all of the solutions you created above,  including a column with the status of the solution versions. Once it is `Active`, your solution is ready to be reviewed. It is also capable of being deployed.\n",
    "\n",
    "Or simply run the cell below to keep track of the solution version creation status."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "At least one solution build is still in progress\n",
      "At least one solution build is still in progress\n",
      "At least one solution build is still in progress\n",
      "At least one solution build is still in progress\n",
      "At least one solution build is still in progress\n",
      "At least one solution build is still in progress\n",
      "At least one solution build is still in progress\n",
      "At least one solution build is still in progress\n",
      "At least one solution build is still in progress\n",
      "At least one solution build is still in progress\n",
      "At least one solution build is still in progress\n",
      "At least one solution build is still in progress\n",
      "At least one solution build is still in progress\n",
      "At least one solution build is still in progress\n",
      "At least one solution build is still in progress\n",
      "At least one solution build is still in progress\n",
      "At least one solution build is still in progress\n",
      "At least one solution build is still in progress\n",
      "At least one solution build is still in progress\n",
      "At least one solution build is still in progress\n",
      "At least one solution build is still in progress\n",
      "At least one solution build is still in progress\n",
      "At least one solution build is still in progress\n",
      "At least one solution build is still in progress\n",
      "At least one solution build is still in progress\n",
      "At least one solution build is still in progress\n",
      "At least one solution build is still in progress\n",
      "At least one solution build is still in progress\n",
      "At least one solution build is still in progress\n",
      "At least one solution build is still in progress\n",
      "At least one solution build is still in progress\n",
      "At least one solution build is still in progress\n",
      "At least one solution build is still in progress\n",
      "At least one solution build is still in progress\n",
      "At least one solution build is still in progress\n",
      "At least one solution build is still in progress\n",
      "Build succeeded for arn:aws:personalize:us-east-1:835319576252:solution/personalize-poc-sims/666f373e\n",
      "At least one solution build is still in progress\n",
      "At least one solution build is still in progress\n",
      "At least one solution build is still in progress\n",
      "Build succeeded for arn:aws:personalize:us-east-1:835319576252:solution/personalize-poc-rerank/44c934a6\n",
      "At least one solution build is still in progress\n",
      "At least one solution build is still in progress\n",
      "At least one solution build is still in progress\n",
      "At least one solution build is still in progress\n",
      "At least one solution build is still in progress\n",
      "At least one solution build is still in progress\n",
      "At least one solution build is still in progress\n",
      "At least one solution build is still in progress\n",
      "At least one solution build is still in progress\n",
      "At least one solution build is still in progress\n",
      "At least one solution build is still in progress\n",
      "At least one solution build is still in progress\n",
      "At least one solution build is still in progress\n",
      "At least one solution build is still in progress\n",
      "At least one solution build is still in progress\n",
      "At least one solution build is still in progress\n",
      "At least one solution build is still in progress\n",
      "At least one solution build is still in progress\n",
      "At least one solution build is still in progress\n",
      "At least one solution build is still in progress\n",
      "At least one solution build is still in progress\n",
      "At least one solution build is still in progress\n",
      "At least one solution build is still in progress\n",
      "Build succeeded for arn:aws:personalize:us-east-1:835319576252:solution/personalize-poc-userpersonalization/d8407cb8\n",
      "CPU times: user 456 ms, sys: 58.3 ms, total: 514 ms\n",
      "Wall time: 1h 2min 8s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "in_progress_solution_versions = [\n",
    "    userpersonalization_solution_version_arn,\n",
    "    sims_solution_version_arn,\n",
    "    rerank_solution_version_arn\n",
    "]\n",
    "\n",
    "max_time = time.time() + 10*60*60 # 10 hours\n",
    "while time.time() < max_time:\n",
    "    for solution_version_arn in in_progress_solution_versions:\n",
    "        version_response = personalize.describe_solution_version(\n",
    "            solutionVersionArn = solution_version_arn\n",
    "        )\n",
    "        status = version_response[\"solutionVersion\"][\"status\"]\n",
    "        \n",
    "        if status == \"ACTIVE\":\n",
    "            print(\"Build succeeded for {}\".format(solution_version_arn))\n",
    "            in_progress_solution_versions.remove(solution_version_arn)\n",
    "        elif status == \"CREATE FAILED\":\n",
    "            print(\"Build failed for {}\".format(solution_version_arn))\n",
    "            in_progress_solution_versions.remove(solution_version_arn)\n",
    "    \n",
    "    if len(in_progress_solution_versions) <= 0:\n",
    "        break\n",
    "    else:\n",
    "        print(\"At least one solution build is still in progress\")\n",
    "        \n",
    "    time.sleep(60)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hyperparameter tuning\n",
    "\n",
    "Personalize offers the option of running hyperparameter tuning when creating a solution. Because of the additional computation required to perform hyperparameter tuning, this feature is turned off by default. Therefore, the solutions we created above, will simply use the default values of the hyperparameters for each recipe. For more information about hyperparameter tuning, see the [documentation](https://docs.aws.amazon.com/personalize/latest/dg/customizing-solution-config-hpo.html).\n",
    "\n",
    "If you have settled on the correct recipe to use, and are ready to run hyperparameter tuning, the following code shows how you would do so, using SIMS as an example.\n",
    "\n",
    "```python\n",
    "sims_create_solution_response = personalize.create_solution(\n",
    "    name = \"personalize-poc-sims-hpo\",\n",
    "    datasetGroupArn = dataset_group_arn,\n",
    "    recipeArn = SIMS_recipe_arn,\n",
    "    performHPO=True\n",
    ")\n",
    "\n",
    "sims_solution_arn = sims_create_solution_response['solutionArn']\n",
    "print(json.dumps(sims_create_solution_response, indent=2))\n",
    "```\n",
    "\n",
    "If you already know the values you want to use for a specific hyperparameter, you can also set this value when you create the solution. The code below shows how you could set the value for the `popularity_discount_factor` for the SIMS recipe.\n",
    "\n",
    "```python\n",
    "sims_create_solution_response = personalize.create_solution(\n",
    "    name = \"personalize-poc-sims-set-hp\",\n",
    "    datasetGroupArn = dataset_group_arn,\n",
    "    recipeArn = SIMS_recipe_arn,\n",
    "    solutionConfig = {\n",
    "        'algorithmHyperParameters': {\n",
    "            'popularity_discount_factor': '0.7'\n",
    "        }\n",
    "    }\n",
    ")\n",
    "\n",
    "sims_solution_arn = sims_create_solution_response['solutionArn']\n",
    "print(json.dumps(sims_create_solution_response, indent=2))\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate solution versions <a class=\"anchor\" id=\"eval\"></a>\n",
    "[Back to top](#top)\n",
    "\n",
    "It should not take more than an hour to train all the solutions from this notebook. While training is in progress, we recommend taking the time to read up on the various algorithms (recipes) and their behavior in detail. This is also a good time to consider alternatives to how the data was fed into the system and what kind of results you expect to see.\n",
    "\n",
    "When the solutions finish creating, the next step is to obtain the evaluation metrics. Personalize calculates these metrics based on a subset of the training data. The image below illustrates how Personalize splits the data. Given 10 users, with 10 interactions each (a circle represents an interaction), the interactions are ordered from oldest to newest based on the timestamp. Personalize uses all of the interaction data from 90% of the users (blue circles) to train the solution version, and the remaining 10% for evaluation. For each of the users in the remaining 10%, 90% of their interaction data (green circles) is used as input for the call to the trained model. The remaining 10% of their data (orange circle) is compared to the output produced by the model and used to calculate the evaluation metrics.\n",
    "\n",
    "![personalize metrics](static/imgs/personalize_metrics.png)\n",
    "\n",
    "We recommend reading [the documentation](https://docs.aws.amazon.com/personalize/latest/dg/working-with-training-metrics.html) to understand the metrics, but we have also copied parts of the documentation below for convenience.\n",
    "\n",
    "You need to understand the following terms regarding evaluation in Personalize:\n",
    "\n",
    "* *Relevant recommendation* refers to a recommendation that matches a value in the testing data for the particular user.\n",
    "* *Rank* refers to the position of a recommended item in the list of recommendations. Position 1 (the top of the list) is presumed to be the most relevant to the user.\n",
    "* *Query* refers to the internal equivalent of a GetRecommendations call.\n",
    "\n",
    "The metrics produced by Personalize are:\n",
    "* **coverage**: The proportion of unique recommended items from all queries out of the total number of unique items in the training data (includes both the Items and Interactions datasets).\n",
    "* **mean_reciprocal_rank_at_25**: The [mean of the reciprocal ranks](https://en.wikipedia.org/wiki/Mean_reciprocal_rank) of the first relevant recommendation out of the top 25 recommendations over all queries. This metric is appropriate if you're interested in the single highest ranked recommendation.\n",
    "* **normalized_discounted_cumulative_gain_at_K**: Discounted gain assumes that recommendations lower on a list of recommendations are less relevant than higher recommendations. Therefore, each recommendation is discounted (given a lower weight) by a factor dependent on its position. To produce the [cumulative discounted gain](https://en.wikipedia.org/wiki/Discounted_cumulative_gain) (DCG) at K, each relevant discounted recommendation in the top K recommendations is summed together. The normalized discounted cumulative gain (NDCG) is the DCG divided by the ideal DCG such that NDCG is between 0 - 1. (The ideal DCG is where the top K recommendations are sorted by relevance.) Amazon Personalize uses a weighting factor of 1/log(1 + position), where the top of the list is position 1. This metric rewards relevant items that appear near the top of the list, because the top of a list usually draws more attention.\n",
    "* **precision_at_K**: The number of relevant recommendations out of the top K recommendations divided by K. This metric rewards precise recommendation of the relevant items.\n",
    "\n",
    "Let's take a look at the evaluation metrics for each of the solutions produced in this notebook. *Please note, your results might differ from the results described in the text of this notebook, due to the quality of the Movielens dataset.* \n",
    "\n",
    "### User Personalization metrics\n",
    "\n",
    "First, retrieve the evaluation metrics for the User Personalization solution version."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"solutionVersionArn\": \"arn:aws:personalize:us-east-1:835319576252:solution/personalize-poc-userpersonalization/d8407cb8\",\n",
      "  \"metrics\": {\n",
      "    \"coverage\": 0.0786,\n",
      "    \"mean_reciprocal_rank_at_25\": 0.2263,\n",
      "    \"normalized_discounted_cumulative_gain_at_10\": 0.2174,\n",
      "    \"normalized_discounted_cumulative_gain_at_25\": 0.2831,\n",
      "    \"normalized_discounted_cumulative_gain_at_5\": 0.1802,\n",
      "    \"precision_at_10\": 0.0464,\n",
      "    \"precision_at_25\": 0.0321,\n",
      "    \"precision_at_5\": 0.0571\n",
      "  },\n",
      "  \"ResponseMetadata\": {\n",
      "    \"RequestId\": \"d88a71cd-8b6a-4096-9862-f58d6bbcb8bd\",\n",
      "    \"HTTPStatusCode\": 200,\n",
      "    \"HTTPHeaders\": {\n",
      "      \"content-type\": \"application/x-amz-json-1.1\",\n",
      "      \"date\": \"Sun, 08 Nov 2020 07:48:19 GMT\",\n",
      "      \"x-amzn-requestid\": \"d88a71cd-8b6a-4096-9862-f58d6bbcb8bd\",\n",
      "      \"content-length\": \"419\",\n",
      "      \"connection\": \"keep-alive\"\n",
      "    },\n",
      "    \"RetryAttempts\": 0\n",
      "  }\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "user_personalization_solution_metrics_response = personalize.get_solution_metrics(\n",
    "    solutionVersionArn = userpersonalization_solution_version_arn\n",
    ")\n",
    "\n",
    "print(json.dumps(user_personalization_solution_metrics_response, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The normalized discounted cumulative gain above tells us that at 5 items, we have less than a (38% for full 22% for small) chance in a recommendation being a part of a user's interaction history (in the hold out phase from training and validation). Around 13% of the recommended items are unique, and we have a precision of only (14% for full, 7.5% for small) in the top 5 recommended items. \n",
    "\n",
    "This is clearly not a great model, but keep in mind that we had to use rating data for our interactions because Movielens is an explicit dataset based on ratings. The Timestamps also were from the time that the movie was rated, not watched, so the order is not the same as the order a viewer would watch movies.\n",
    "\n",
    "### SIMS metrics\n",
    "\n",
    "Now, retrieve the evaluation metrics for the SIMS solution version."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"solutionVersionArn\": \"arn:aws:personalize:us-east-1:835319576252:solution/personalize-poc-sims/666f373e\",\n",
      "  \"metrics\": {\n",
      "    \"coverage\": 0.1843,\n",
      "    \"mean_reciprocal_rank_at_25\": 0.1777,\n",
      "    \"normalized_discounted_cumulative_gain_at_10\": 0.2299,\n",
      "    \"normalized_discounted_cumulative_gain_at_25\": 0.2962,\n",
      "    \"normalized_discounted_cumulative_gain_at_5\": 0.1954,\n",
      "    \"precision_at_10\": 0.0588,\n",
      "    \"precision_at_25\": 0.0418,\n",
      "    \"precision_at_5\": 0.0824\n",
      "  },\n",
      "  \"ResponseMetadata\": {\n",
      "    \"RequestId\": \"48ac3d87-8481-41ec-8b9e-8725241f0ba0\",\n",
      "    \"HTTPStatusCode\": 200,\n",
      "    \"HTTPHeaders\": {\n",
      "      \"content-type\": \"application/x-amz-json-1.1\",\n",
      "      \"date\": \"Sun, 08 Nov 2020 07:48:20 GMT\",\n",
      "      \"x-amzn-requestid\": \"48ac3d87-8481-41ec-8b9e-8725241f0ba0\",\n",
      "      \"content-length\": \"404\",\n",
      "      \"connection\": \"keep-alive\"\n",
      "    },\n",
      "    \"RetryAttempts\": 0\n",
      "  }\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "sims_solution_metrics_response = personalize.get_solution_metrics(\n",
    "    solutionVersionArn = sims_solution_version_arn\n",
    ")\n",
    "\n",
    "print(json.dumps(sims_solution_metrics_response, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this example we are seeing a slightly elevated precision at 5 items, a little over (4.5% for full, 6.4% for small) this time. Effectively this is probably within the margin of error, but given that no effort was made to mask popularity, it may just be returning super popular results that a large volume of users have interacted with in some way. \n",
    "\n",
    "### Personalized ranking metrics\n",
    "\n",
    "Now, retrieve the evaluation metrics for the personalized ranking solution version."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"solutionVersionArn\": \"arn:aws:personalize:us-east-1:835319576252:solution/personalize-poc-rerank/44c934a6\",\n",
      "  \"metrics\": {\n",
      "    \"coverage\": 0.0038,\n",
      "    \"mean_reciprocal_rank_at_25\": 0.0593,\n",
      "    \"normalized_discounted_cumulative_gain_at_10\": 0.0824,\n",
      "    \"normalized_discounted_cumulative_gain_at_25\": 0.0958,\n",
      "    \"normalized_discounted_cumulative_gain_at_5\": 0.0377,\n",
      "    \"precision_at_10\": 0.0189,\n",
      "    \"precision_at_25\": 0.0113,\n",
      "    \"precision_at_5\": 0.0075\n",
      "  },\n",
      "  \"ResponseMetadata\": {\n",
      "    \"RequestId\": \"d81e90b9-d502-465f-a234-eeec9fa22341\",\n",
      "    \"HTTPStatusCode\": 200,\n",
      "    \"HTTPHeaders\": {\n",
      "      \"content-type\": \"application/x-amz-json-1.1\",\n",
      "      \"date\": \"Sun, 08 Nov 2020 07:48:20 GMT\",\n",
      "      \"x-amzn-requestid\": \"d81e90b9-d502-465f-a234-eeec9fa22341\",\n",
      "      \"content-length\": \"406\",\n",
      "      \"connection\": \"keep-alive\"\n",
      "    },\n",
      "    \"RetryAttempts\": 0\n",
      "  }\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "rerank_solution_metrics_response = personalize.get_solution_metrics(\n",
    "    solutionVersionArn = rerank_solution_version_arn\n",
    ")\n",
    "\n",
    "print(json.dumps(rerank_solution_metrics_response, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just a quick comment on this one, here we see again a precision of near (2.7% for full, 2.2% for small), as this is based on User Personalization, that is to be expected. However the sample items are not the same for validaiton, thus the low scoring.\n",
    "\n",
    "## Using evaluation metrics <a class=\"anchor\" id=\"use\"></a>\n",
    "[Back to top](#top)\n",
    "\n",
    "It is important to use evaluation metrics carefully. There are a number of factors to keep in mind.\n",
    "\n",
    "* If there is an existing recommendation system in place, this will have influenced the user's interaction history which you use to train your new solutions. This means the evaluation metrics are biased to favor the existing solution. If you work to push the evaluation metrics to match or exceed the existing solution, you may just be pushing the User Personalization to behave like the existing solution and might not end up with something better.\n",
    "* The HRNN Coldstart recipe is difficult to evaluate using the metrics produced by Amazon Personalize. The aim of the recipe is to recommend items which are new to your business. Therefore, these items will not appear in the existing user transaction data which is used to compute the evaluation metrics. As a result, HRNN Coldstart will never appear to perform better than the other recipes, when compared on the evaluation metrics alone. Note: The User Personalization recipe also includes improved cold start functionality\n",
    "\n",
    "Keeping in mind these factors, the evaluation metrics produced by Personalize are generally useful for two cases:\n",
    "1. Comparing the performance of solution versions trained on the same recipe, but with different values for the hyperparameters and features (impression data etc)\n",
    "1. Comparing the performance of solution versions trained on different recipes (except HRNN Coldstart).\n",
    "\n",
    "Properly evaluating a recommendation system is always best done through A/B testing while measuring actual business outcomes. Since recommendations generated by a system usually influence the user behavior which it is based on, it is better to run small experiments and apply A/B testing for longer periods of time. Over time, the bias from the existing model will fade."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Storing useful variables <a class=\"anchor\" id=\"vars\"></a>\n",
    "[Back to top](#top)\n",
    "\n",
    "Before exiting this notebook, run the following cells to save the version ARNs for use in the next notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Stored 'userpersonalization_solution_version_arn' (str)\n",
      "Stored 'sims_solution_version_arn' (str)\n",
      "Stored 'rerank_solution_version_arn' (str)\n",
      "Stored 'user_personalization_solution_arn' (str)\n",
      "Stored 'sims_solution_arn' (str)\n",
      "Stored 'rerank_solution_arn' (str)\n"
     ]
    }
   ],
   "source": [
    "%store userpersonalization_solution_version_arn\n",
    "%store sims_solution_version_arn\n",
    "%store rerank_solution_version_arn\n",
    "%store user_personalization_solution_arn\n",
    "%store sims_solution_arn\n",
    "%store rerank_solution_arn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Release Resources"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%javascript\n",
    "Jupyter.notebook.save_checkpoint();\n",
    "Jupyter.notebook.session.delete();"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
